Analyzing my Spotify listening history 🎵 - Part 3
description
We need to retrieve the audio features for our songs. For this, we use the song IDs we retrieved in part 1. Lets first define our spotify API credentials again, an initialize the library we use to interact, Spotipy.
#collapse-show
SPOTIFY_API_KEY = os.getenv('SPOTIFY_API_KEY')
SPOTIFY_CLIENT_ID = os.getenv('SPOTIFY_CLIENT_ID')
spotify_search_url = 'https://api.spotify.com/v1/search?q={q}&type={type}'
spotify_headers = {"Accept": "application/json", "Authorization":"Bearer " + SPOTIFY_API_KEY, "Content-Type": "application/json"}
client_credentials_manager = SpotifyClientCredentials(SPOTIFY_CLIENT_ID, SPOTIFY_API_KEY)
spotify = spotipy.Spotify(client_credentials_manager=client_credentials_manager)
Let's load up the track IDs and put them in a list, for easier access.
#collapse-show
def get_track_info():
lines = []
for i in json_lines.reader(open('data/EndSong.json', encoding='utf-8')):
lines.append(i)
df = pd.DataFrame(lines)
df = df.drop(['username', 'user_agent_decrypted', 'incognito_mode', 'platform', 'ip_addr_decrypted', 'region', 'longitude', 'latitude', 'city'], axis=1)
df.ts = pd.to_datetime(df.ts)
return df
track_info = get_track_info()
track_ids = pd.read_csv('track_ids.csv')
track_ids = track_ids.values.reshape(-1).tolist()
top_1_genres = pd.read_csv('top_1_genres.csv')
top_1_genres_filtered = pd.read_csv('top_1_genres_filtered.csv')
We can utilize the .audio_features call (docs) to retrieve a lot of information about a track. We give this call a Spotify track URI, so Spotify knows exactly which track the request is about. However, some songs did not give us an URI when we were getting those. So the resulting track IDs have some None in them.
Lets see what the audio features are of Blue from Gemini.
spotify.audio_features([track_ids[2]])
It's important to know what these features actually represent. You can find the full list on the Spotify website.
| Key | Value Type | Value Description |
|---|---|---|
| duration_ms | int | The duration of the track in milliseconds. |
| key | int | The estimated overall key of the track. Integers map to pitches using standard Pitch Class notation . E.g. 0 = C, 1 = C♯/D♭, 2 = D, and so on. If no key was detected, the value is -1. |
| mode | int | Mode indicates the modality (major or minor) of a track, the type of scale from which its melodic content is derived. Major is represented by 1 and minor is 0. |
| time_signature | int | An estimated overall time signature of a track. The time signature (meter) is a notational convention to specify how many beats are in each bar (or measure). |
| acousticness | float | A confidence measure from 0.0 to 1.0 of whether the track is acoustic. 1.0 represents high confidence the track is acoustic. |
| danceability | float | Danceability describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity. A value of 0.0 is least danceable and 1.0 is most danceable. |
| energy | float | Energy is a measure from 0.0 to 1.0 and represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy. For example, death metal has high energy, while a Bach prelude scores low on the scale. Perceptual features contributing to this attribute include dynamic range, perceived loudness, timbre, onset rate, and general entropy. |
| instrumentalness | float | Predicts whether a track contains no vocals. “Ooh” and “aah” sounds are treated as instrumental in this context. Rap or spoken word tracks are clearly “vocal”. The closer the instrumentalness value is to 1.0, the greater likelihood the track contains no vocal content. Values above 0.5 are intended to represent instrumental tracks, but confidence is higher as the value approaches 1.0. |
| liveness | float | Detects the presence of an audience in the recording. Higher liveness values represent an increased probability that the track was performed live. A value above 0.8 provides strong likelihood that the track is live. |
| loudness | float | The overall loudness of a track in decibels (dB). Loudness values are averaged across the entire track and are useful for comparing relative loudness of tracks. Loudness is the quality of a sound that is the primary psychological correlate of physical strength (amplitude). Values typical range between -60 and 0 db. |
| speechiness | float | Speechiness detects the presence of spoken words in a track. The more exclusively speech-like the recording (e.g. talk show, audio book, poetry), the closer to 1.0 the attribute value. Values above 0.66 describe tracks that are probably made entirely of spoken words. Values between 0.33 and 0.66 describe tracks that may contain both music and speech, either in sections or layered, including such cases as rap music. Values below 0.33 most likely represent music and other non-speech-like tracks. |
| valence | float | A measure from 0.0 to 1.0 describing the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g. happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g. sad, depressed, angry). |
| tempo | float | The overall estimated tempo of a track in beats per minute (BPM). In musical terminology, tempo is the speed or pace of a given piece and derives directly from the average beat duration. |
If you wan't to see how we retrieve this information, please check the collapsed cell below.
#collapse-hide
track_features_list = []
import math
for i in tqdm(range(len(track_ids))):
if track_ids[i] == '':
track_features_temp = None
if isinstance(track_ids[i], float):
if math.isnan(track_ids[i]):
print('isnan')
track_features_temp = None
else:
try:
track_features_temp = spotify.audio_features(track_ids[i])
except Exception as e:
print(e)
print(track_ids[i])
track_features_list += track_features_temp if track_features_temp else [{}]
track_features = pd.DataFrame(track_features_list)
We combine the data, resulting in a single dataframe that holds all our data.
# Add our audio features with our track information
comb = pd.concat([track_info, track_features], axis=1)
# Also add the top genres of each song
comb['top_1_genres'] = top_1_genres.top_1_genres
comb['top_1_genres_filtered'] = top_1_genres_filtered.top_1_genres_filtered
# We can just choose one of the audio features and drop on that, since each song either has all of the features or none.
comb = comb.dropna(subset=['acousticness'])
audio_columns = [
'acousticness',
'danceability',
'energy',
'instrumentalness',
'liveness',
'loudness',
'speechiness',
'tempo',
'valence',
'duration_ms'
]
The Spotify website also includes distribution plots for most features, so we will take at how you might differ from the general plots and how you can see what causes this. I have selected a subset of audio features, but you can evaluate the rest yourself. Looking at the values they report, their analysis seems to have been done on about 10k data points. To get a better comparison, we will sample 10k data points and compare with that. However, when diving deeper into something interesting, we use the full dataset.
#collapse-hide
sample_10k = comb.sample(10000, random_state=42)
We can see there is little to no acousticness after 0.6 in my listening. This is very expected, with both hip hop and EDM containing none or little acoustic elements.
Danceability
Danceability describes how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity. A value of 0.0 is least danceable and 1.0 is most danceable.
Here, we take a look at danceability. Given my main genres, we should see higher danceability than normally, with EDM being a big part of my listening. And indeed we do see this. We have quite some peaks around 0.8, while the general distribution is closing in on 0 around 0.8. Since we only have an image, we cannot compare means, but I expect the general distribution mean to be somewhere between 0.5-0.6, while mine is 0.67
Energy
Energy is a measure from 0.0 to 1.0 and represents a perceptual measure of intensity and activity. Typically, energetic tracks feel fast, loud, and noisy. For example, death metal has high energy, while a Bach prelude scores low on the scale. Perceptual features contributing to this attribute include dynamic range, perceived loudness, timbre, onset rate, and general entropy.
Now, energy is a bit strange. In my year in review, Spotify always telling me I listen to so much energetic music. However, looking at this plot, I seem to be way lower than the general distribution.
Let's see if we get some more clarity if we look at the energy per artist.
These results I find quite surprising. Yellow Claw, a Dutch house group whom I would consider to be generally pretty high energy is equal or below quite some hip hop artists, like Mac Miller and Kid Cudi. We can see the maximum energy in here is 6ix9ine, with a solid 0.69 😂. There is no denying that 6ix9ine is at that level of energy. However, KIDS SEE GHOSTS, the front for Kid Cudi and Kanye West, somehow also has 0.69, while I'd say that is much lower in energy. If we look at Reborn, the most popular song on the album, it's clearly a pretty slow song.
Alternatively, we can look at the energy of my top songs. Here we get a clearer picture of the effect of different songs.
There are some pretty interesting facts here. First of, we can see see that my idea of Yellow Claw being very high energy is fairly debunked, at least with Spotify's definition of energy. Because some of these values are very strange, even more so when compared to some other artists and songs. Take for example the song without you from Yellow Claw. This is a dubstep/brostep EDM song, with a pretty high tempo but has a value of 0.4880 for its energy level. This is lower than Frank Sinatra had on average in the previous table. In my opinion, something is wrong there. Furthermore, we can also see that their song Stacks has an energy value of 0.0812, which, by all accounts, should be an anomaly. My expectation is that they trained a neural network with some hand labeled songs, and then applied that to all songs to estimate these values, and something went wrong in the case of this song.
Instrumentalness
Predicts whether a track contains no vocals. “Ooh” and “aah” sounds are treated as instrumental in this context. Rap or spoken word tracks are clearly “vocal”. The closer the instrumentalness value is to 1.0, the greater likelihood the track contains no vocal content. Values above 0.5 are intended to represent instrumental tracks, but confidence is higher as the value approaches 1.0.
So the instrumentalness
This is potentially interesting, since, naturally, hip-hop has a lot of vocals (so low instrumentalness), but genres like electro house and EDM generally don't have many. However, we can quickly see why by looking at the most influential artists for the largest genres. I'm showing the top 2 most played artists per genre, and they are categorized as the most general genre they have.
As most influential EDM artists, we see Yellow Claw and Flume, both of which have a lot of vocals in their music. This explains our instrumentalness plot!. We can also see that the top genres quickly fall apart after rap and EDM, since songs that are tagged with genres like pop rap and hip hop are in many cases also tagged as rap. As a consequence, we get songs with a very small listening share in this overview, like Jason Derulo (8 plays) and One Republic (6 plays). Neither of which I'd generally consider to be my taste.
However, if we redo this analysis, but we drop the most general genres, we get much more interesting results. To be specific, I've ignored the values pop, edm, rap, pop rap and hip hop, unless there were no other genre labels. In that case, we still take that genre. The result is below, and quite interesting. The table shows the top 20 most played artists and the genre they fall in.
Without the super general genres, we can finally start to see some trends. Given the fact that rap is still the main genre of 5 artists, this often seems to be one of few labels applied, possibly with the other labels I was removing. The results is that these artists do not belong to any specific subgenres within hip hop, which is fairly cool to see. Does this also mean they are per definition mainstream?
loudness
The overall loudness of a track in decibels (dB). Loudness values are averaged across the entire track and are useful for comparing relative loudness of tracks. Loudness is the quality of a sound that is the primary psychological correlate of physical strength (amplitude). Values typical range between -60 and 0 db.
We can see my music is quite a bit louder than average, with almost all of the mass of the distribution being between -10 and 0, with a small tail between -15 and -10. In the general distribution, the tail extends quite a bit beyond -20, whereas there is virtually nothing beyond -18 in my distribution. Let's see who is responsible for all of this noise 😠.
This table is pretty clear. We see that none of my top 20 most played artists have an average loudness lower than -10, indicating quite high loudness on average. If we look at the top 5, we see that apparently hip hop artists are very loud, but also specifically Kid Cudi, Kanye West, and then those two combined in KIDS SEE GHOSTS. The reason for this I expect to be that they have sounds playing at almost at all times, rather than that they are extremely loud in their peaks.
It also comes as no surprise that 6ix9ine is the loudest, since he's essentially screaming in most of his songs 😅.
valence
A measure from 0.0 to 1.0 describing the musical positiveness conveyed by a track. Tracks with high valence sound more positive (e.g. happy, cheerful, euphoric), while tracks with low valence sound more negative (e.g. sad, depressed, angry).
This is a very interesting feature. Let's see what information it gives us!
That's quite the difference! First off, we can see a clear decline towards 1.0 for my distribution, whereas in the general distribution, this is much rounder. Furthermore, we can see that my distribution a lot spikier, possibly indicating the effects of certain albums/songs that have been played a lot and have a very narrow spread with regards to valence. In general we can see my music taste leans more towards the low side of valence, indicating a preference for sad, depressed and angry music. In general, I think it will be angry, which is the sentiment in a lot of hip hop, but also techno, drum and bass and some parts of house.
If we look at the most positive and most negative songs with more than 5 plays, we see some interesting results.
We can see that there are likely some errors in here, with White Crime and White America both having a score of 0.0. On the positive side of the scale (close to 1), we see Ed Sheeran which is not surprising. We also see Yung Internet with Helemaal Top, which is a song about feeling great. However, we also have Kendrick Lamar's Momma. Now, this is interesting cause it does not typically resonate as a very happy or cheerful song. We'll attribute this to the Spotify interpretation of this value. See for yourself:
Some interesting insights:
- Kanye is depressing: Kanye West is very low in valence, meaning most of his songs are angry, depressed or sad. This makes sense, given his oeuvre, with songs like Waves (sad), Violent Crimes (sad), Piss On Your Grave (angry) and I Am A God (angry).
- Bias of electronic music: We have some electronic artists like Tchami and Netsky which also rank very low, but which are not specifically angry or sad music producers in my experience. Maybe electronic music has a bias here and is faster to be considered angry or sad?
- Non-polarity: We see quite some artists hovering around 0.5, indicating either a healthy balance in valence between their songs or just a general non-polarity in their songs. I took a detailed look at Kanye West for the first point, and he has a fairly wide spread, with the weight more on the sad and angry side, hence his low average valence. I assume most artists will be similar, and have a wide spread between their songs.